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ABSTRACT 



The purpose of this study was to adapt and illustrate the use of a 
computer program to score binary patterns of response on a short- 
form predictor test (Electronics Technician Selection Test and the 
General Classification Test) so as to maximize the correlation 
between this predictor and a criterion (tlie final school grade in the 
Basic Electronics and Electricity School). 



2 



TABLE OF CONTENTS 



I. INTRODUCTION 9 

II. BACKGROUND 10 

III. THE PROBLEM 11 

IV. DATA 14 

A. DATA PREPARATION 15 

V. THE MODEL - - ---17 

A. DOUBLE PRECISION REQUIREMENT 17 

B. THE DATA CARD - 17 

C . READING THE DATA 18 

D. THE JOINT FREQUENCY DISTRIBUTION 20 

E. COMPUTATION OF PATTERN SCORES -20 

F. ASSIGNMENT OF PATTERN SCORES TO SUBJECTS ---22 

G. COMPUTATION OF CORRELATIONS 22 

H. CONSTRUCTION OF RESPONSE PATTERNS 23 

I. MULTIPLE CORRELATION COEFFICIENT -23 

J. COMPUTATION OF REGRESSION WEIGHTS 24 

K. OUTPUT I --25 

L. OUTPUT II - 25 

VI. CROSS VALIDATION 26 

A. ASSIGNMENT OF PATTERN SCORES - - 27 

VII. METHOD 28 



3 



ayK-pi^ 








A. PRELIMINARY STUDIES - 28 

B. ASSIGNMENT OF PATTERN SCORES 31 

1. The First Solution 31 

2. The Second Solution 7 32 

3. The Third Solution 32 

4. The Final Solution 32 

C. READ IN OF ALTERNATE DATA 33 

D. THE ETST STUDY 33 

1. Test- Retest Reliability Coefficient -34 

2. Correction for Attenuation 34 

VIII. RESULTS - 36 

A. DETERMINATION OF LINEARITY- 36 

B. CORRELATION COEFFICIENTS AND TEST 

STATISTICS 36 

IX. CONCLUSION- 41 

APPENDIX A SEVENBESTGCT ITEMS SELECTED BY "SEQUIN” --42 

APPENDIX B SEVEN BEST ETST ITEMS SELECTED BY 

"SEQUIN” - - 43 

APPENDIX C FIRST DATA PREPARATION PROGRAM 44 

APPENDIX D LISTING OF FIRST DATA PREPARATION 

PROGRAM 45 

APPENDIX E FLOWCHART OF SECOND DATA PREPARATION 

PROGRAM -46 

APPENDIX F LISTING OF SECOND DATA PREPARATION 

PROGRAM - 47 

APPENDIX G LISTING OF VALIDATION PROGRAM 48 

APPENDIX H OUTPUT I: PATTERN INFORMATION 52 



4 



APPENDIX I OUTPUT II: SUBJECT INFORMATION 56 

APPENDIX J CROSS-VALIDATION PROGRAM LISTING 57 

APPENDIX K OUTPUT III: SUBJECT INFORMATION (CROSS- 

VALADATION) 61 

APPENDIX L GLOSSARY OF COMPUTER TERM VARIABLES 62 

LIST OF REFERENCES 65 

INITIAL DISTRIBUTION LIST 66 

FORM DD 1473 68 






5 



LIST OF TABLES 



I. RECORD DATA CARD SETUP FOR VALIDATION 

AND CROSS-VALIDATION PROGRAMS 19 

II. CORRELATION COEFFICIENTS AND TEST 

STATISTICS DERIVED FROM THE GENERAL 
CLASSIFICATION TEST 30 

III. CORRELATION COEFFICIENTS AND TEST STATISTICS 

DERIVED FROM THE ELECTRONICS TECHNICIAN 
SELECTION TEST 39 



6 



LIST OF FIGURES 



1. CONSTRUCTION OF A MATRIX DESCRIBING THE 

JOINT FREQUENCY DISTRIBUTION 21 

2. SCATTER PLOT OF FINAL SCHOOL GRADE VS. SCORE 

ON FULL ETST ---t 37 

3. SCATTER PLOT OF FINAL SCHOOL GRADE VS. PATTERN 

SCORE ON SHORT- FORM ETST 38 



7 



ACK NOWLEDGE ME NTS 



I wish to thank the following people for their assistance in the 
research for this thesis: Dr. R. A. Weitzman of the Naval 
Postgraduate School, Monterey, California, for his advice and 
guidance; Mr. Leonard Swanson of the Naval Personnel and Training 
Research Laboratory, San Diego, California, for the data; and my 
wife Monika for her understanding during my long and late hours at 
the computer center. 



1. INTRODUCTION 



The Navy has been very much interested in recent years in the 
possibility of using short- form tests to reduce testing time while 
maintaining or even increasing test reliability and validity. 

The advantages of a short- form test are manifold. With a 
short but reliable and valid test the Navy could save thousands of 
dollars in training costs by weeding out, before training even began, 
those individuals who v/ould probably not succeed. The administration 
of the test could be done at a training command, e. g. , Naval Training 
Center, San Diego, Bainbridge, etc. , or even by a recruiter. For 
example, if an individual desires to be a radioman and talks to a 
recruiter about joining the Navy only if accepted for radioman training, 
it would be advantageous for both the service and that individual if a 
brief test of possibly five to seven minutes' duration could be admin- 
istered, graded and evaluated on the spot against the individual's 
desires for such a Navy career. With this brief test both the Navy 
and the potential recruit would know, in a relatively short period 
of time, whether the man would succeed in radioman training. 
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II. BACKGROUND 



Moonan (Ref. 1) pioneered this type of work for the Navy by 
constructing a computer program having the capability of identifying 
combinations of test items that have maximal validity. This program, 
entitled SEQUIN (an acronym for Seq uential Item Nominator) first 
selects an item that has highest validity with the criterion. The pro- 
gram then continues to select another item which, when. combined 
with the first, produces a two- item test with a higher validity than 
any other two-item test that includes the first item. This process 
continues until the required number of items is selected and the 
maximum validity for this number of items is obtained. The advan- 
tage of such a program is that a fairly long test, such as the General 
Classification Test (GCT), might be shortened without sacrificing 
validity while test time might be significantly reduced. 

SEQUIN has shown, repeatedly, that a short- form test is at 
least as predictive of final school grade as its long- form counterpart 
(Ref. 2). Swanson and Rimland (Ref. 3) have found that a short form 
of the GCT, e. g. one- half to one- third of the original length, is even 
more predictive of recruit final achievement (RFAT) than the complete 
form. 
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III. THE PROBLEM 



This study attempts to increase further the predictive validity 
of an already brief test. The method, developed by Dr. R. A. 
Weitzman of the Naval Postgraduate School, Monterey, California, 
is to weight item responses so as to maximize the correlation with the 
criterion. 

On an n-item test where each question is graded to be either 
correct or incorrect, there are 2^^ different possible patterns of 
correct and incorrect responses. Thus, for example, on a five-item 
test there are 32 possible pattern scores as opposed to six possible 
scores if just the number of correct responses were tallied. 

For example, suppose a three question test is given to a group 
of recruits in an attempt to predict their success in a Navy training 

o 

school. There are eight (2 ) combinations of patterns running from 
000 to 111 (where zeros are incorrect responses and ones are correct). 
A subject having a pattern of 101 has the same number correct as 
another subject with the pattern 110, that is, two out of three. 

However, the first individual’s score might be more predictive of 
success in a particular training school than the second subject's 
binary pattern. 

This study will focus on four different tests or test scores, 
defined as follows; 
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1. Predictor - the predictor is a long- form test used for 
predicting success in a Navy training school. In this study, the 
predictor is the Electronics Technician Selection Test or the General 
Classification Test. Scores on the predictor are determined by 
counting the items answered correctly. 

2. Criterion - the final school grade in the Basic Electronics 
and Electricity School. 

3. Total Correct - the total number of correct responses out 
of the seven questions selected by SEQUIN analysis for this study. 

4. Pattern Score - a special score assigned to each pattern of 
responses on the same seven items used to compute total correct. 

(A precise definition of pattern scores will be given in Section VE. ) 

Thus, the purpose of this study was to: 

1. Gather large pools of data from a Navy training school, 

2. Extract several suitable questions from the General Classi- 
fication Test (GCT) and the Electronic Technician Selection Test 
(ETST), 

3. Write a computer program that ; 

a. constructs all possible patterns of ones and zeros for 
the number of extracted questions 

b. calculates pattern scores for each individual pattern 

c. assigns pattern scores to subjects 

d. correlates the pattern scores of the subjects with 
their final school grades 
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e. correlates the standard predictor test scores (either 
GCT or ETST) with final school grades 

f. correlates total correct, with final school 
grades 

g. correlates pattern score with total correct 

h. calculates a multiple correlation coefficient between 
a combination of pattern scores and total correct and 
final school grades 

i. calculates test statistics for the correlations 

j. calculates regression weights for predicting final 
school grades from total-correct scores 

k. creates a frequency distribution showing number of 
subjects with each pattern score 

l. outputs all information in an easy- to- read form for 
use in fitaire studies 

4. Determine those patterns indicative of success for a parti- 
cular training school, 

5. Test pattern- score predictions by suitable cross-validation. 
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IV. DATA 



All data used in this research were obtained from Mr. Leonard 
Swanson of the Naval Personnel and Training Research Laboratory, 

San Diego, California, and were stored on nine- track magnetic tape 
(Ref. 4). The data consisted of the individual records of approxi- 
mately 2400 trainees who started, but not necessarily finished, the 
Navy Basic Electronics and Electricity School in San Diego. Each 
trainee's record consisted of the equivalent of six-computer card 
records listing such information as: 

1. Responses to items on the GCT, ETST, and Arithmetic 
Aptitude Test (ARI) 

2. Scores on the GCT, ARI, and ETST 

3. Navy service number 

4. Enlisted rating 

5. Final school grade in Basic Electronics and Electricity 
School 

Tests used as predictors included the GCT and ETST. The GCT 
consists of 60 verbal analogies and 40 sentence- completion items 
with a 35- minute time limit. The ETST consists of three separately 
timed sections: math with 20 items and a 25- minute time limit; 
science with 20 items and a 15- minute time limit; and electricity and 
radio with 30 items and a 20- minute time limit (Ref. 5). 
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Tv\'o sets of questions were provided by Mr. Swanson along 
witli their answer keys. The first set of questions, consisting of 
seven GCT items, and the second set, consisting of seven ETST 
items, were selected using the SEQUIN program, Tlie p- values, 
question types and item validities are shown in Appendixes A and B. 

The criterion consisted of final school grades in the Basic 
Electronics and Electricity School. 

A. DATA PREPARATION 

Two programs were written to extract and put into usable forms 
all pertinent data for the study. (A glossary of terms used in all 
programs is contained in Appendix L. ) The first program checked 
for completeness of an individual's record, i.e. , the presence of six 
computer- card images, and rejected those subjects whose files were 
deficient. Unfortunately several records contained special characters, 
e. g. , dashes, asterisks, etc. , instead of integers. Therefore, the 
first data preparation program converted any of these special 
characters to zeros. Thus, a response other than an integer from one 
to five was changed to a zero and counted as an incorrect response. 

If a needed score such as the GCT, ETST or final school grade was 
blank or contained some non- numerical mark, the record for that 
individual was rejected as being incomplete. (It is possible that some 
of those incomplete records resulted from subjects not finishing the 
school, .i. e. , being r,equired to leave the service because of physical. 
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emotional, or academic problems. ) Tlie output from this program was 
written on tape or data cell and on paper. 

Appendix C is the flowchart of this first program. Appendix D 
is the program listing. 

Although a subject’s record consisted of six computer-card 
records, most information was superfluous. Of die six cards, data 
from three, at most, were considered. Using the answer key supplied 
by Mr. Swanson, the second program graded, on different occasions 
those ETST or GOT questions under consideration. Specifically, it 
assigned a value of one to a correct response and a value of zero to 
an incorrect response. By assigning ones and zeros to the responses, 
the binary pattern was formed. The program also read the criterion 
score and the predictor score. 

The output from this second program consisted of: 

1. binary pattern 

2. criterion score 

3. predictor score 

4. an in-house identification number 

5. the subject’s service number 

Appendix E is the flowchart for this second program. Appendix F is 
the program listing. 
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V. THE MODEL 



The main program is divided into several distinct sections: 
reading of data, determination of a joint frequency distribution, 
computation of pattern scores, assignment of pattern scores to sub- 
jects, computation of correlation coefficients (r's), computation of 
test statistics for r differences, construction of response patterns, 
ordering of response patterns according to the scores computed for 
them, calculation of a multiple correlation coefficient, calculation of 
the correlation coefficient between pattern scores and total correct 
construction of a frequency distribution showing the number 
of people with each pattern score, and output (printed and 
punched). 

A complete listing of the program is presented in Appendix G. 

A. DOUBLE PRECISION REQUIREMENT 

Because of the large sample sizes and the relatively large 
magnitude cf several parameters, it was necessary to use double 
precision floating point numbers. 

B. THE DATA CARD 

The data card initializes four variables that are frequently used 
in counting loops (DO loops). The variables, Nl, N2, N3, and N4, 
represent, respectively, the number of people in the sample, number 
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of elements in a pattern, the range of criterion scores, and the 
number of possible binary combinations using N2 items 

C. READING THE DATA 

Data is read in only a prescribed format. For this program, 
the individual's data record card is set up as shown in Table 1. 

There are two read statements. One read statement carries 
out the reading of data that is to be used in the computations of the 
program. Tlie other read statement reads a dummy variable, "IDUM. " 
By placing the read statement involving IDUM before or following the 
main read statement (involving binary pattern, criterion, predictor, 
etc. ),' control over alternate selections of data can be attained. For 
example, if odd numbered data were only to be considered, the read 
statement involving IDUM would follow the main read statement thus 
acting as a dummy procedure to control data input. Note that all 
input data is in FORTRAN " I format. " 

The variable "J" is used as the DO LOOP counter involving 
personnel with only one exception. That exception is in the determi- 
nation of the joint frequency distribution. An "I" DO LOOP is used 
for all other counting operations. 

At this point, the total correct out of the extracted questions 
is calculated. The "E" array stores this information. This array is 
used in the calculation of the sum of total correct for all subjects and 
the sum of the squares of total correct for all subjects. This infor- 
mation is later used in the computation of correlation coefficients 
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TABLE I 



RECORD DATA CARD SETUP FOR 
VALIDATION AND CROSS-VALIDATION PROGRAMS 



Column Number 


Item 


Program Symbol 


1-7 


Subject's binary 
pattern 


P(I,J) 


8,9 


Criterion Score 


C(J) 


10,11 


Predictor 
(Score on ETST) 


A(J) • 


12-15 


In-house ident. 
number 


D(J) 
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and the mean and standard deviation for use in the calculation of a 
test- retest correlation coefficient. 

D. THE JOINT FREQUENCY DISTRIBUTION 

A joint frequency distribution is constructed using decimal 
equivalents of the 2^2 binary patterns and the range of criterion 
scores. The rows of the matrix (denoted by matrix variable F) 
represent the decimal equivalents of the binary patterns, and in tliis 
case there are 128 (2^) binary patterns (the reason for using seven 
questions is explained in the METHODS section). However, the lowest 
binary pattern score (0000000) is also equal to the decimal value zero. 
Therefore, a value of one is added to all decimal equivalents. In 
this way the first row is row one, not zero, and the last row is row 
128. 

The column numbers correspond to successive criterion scores. 

•V 

Column one of the matrix corresponds to the subjects' lowest crite- 
rion score. In this case, the lowest criterion score was 30 and the 
higest was 99. The matrix is represented in Figure 1. 

The "B" array is used to store the decimal equivalent of an ■ 
individual's binary pattern. 

E. COMPUTATION OF PATTERN SCORES 

The pattern score for a pattern is the average score of subjects 
who have the pattern and is calculated from the F matrix by tallying 
the number of subjects having the pattern and each criterion score. 
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Figure 1 



CONSTRUCTION OF A MATRIX DESCRIBING THE 
JOINT FREQUENCY DISTRIBUTION 



Columns** 

(30) (99) 

1 69 



Rows* 



*Row numbers are decimal equivalents of binary patterns plus one. 

**Column numbers are criterion scores plus one minus the lowest 

criterion score. Numbers in parentheses are actual criterion 
scores. 
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This number is multiplied by the criterion score and summed, and 
the sum, SI, is divided by the total number of subjects having the 
pattern, S2. 

If any of the 128 patterns is not used, because no one has the 
pattern, both SI and S2 are set equal to zero, and an arbitrary 
score of -1 is assigned to the pattern. 

Immediately following the computation of all pattern scores, 
the scores are outputted on punched cards. The pattern scores 
obtained in this study are presented in Appendix H. 

F. ASSIGNMENT OF PATTERN SCORE TO SUBJECTS 

A subject's decimal equivalent to his binary pattern is deter- 
mined, and he is assigned the pattern score for that decimal 
equivalent (the row index corresponding to the pattern in the F 
matrix). 

G. COMPUTATION OF CORRELATIONS 

Correlation coefficients are then calculated between the 
criterion and the predictor (GCT or ETST) and between the criterion 
and the assigned pattern scores. 

The sums of criterion scores (Cl), pattern scores (XI), and 
predictor scores (Al) are determined along with the corresponding 
sums of squares (C2, X2, A2). The sum of the products of the 
criterion and predictor scores (V), as well as the criterion and pattern 
scores (W), is also determined. The correlation coefficients for 
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pattern vs. criterion (R2) and predictor vs. criterion (Rl) are then 
calculated. The Z test statistic for the difference between these r’s 
is also calculated. 

Three other correlation coefficients are computed later in the 
program: a multiple correlation coefficient (see I below), the 
correlation coefficient between total-correct and pattern scores, and 
a test- retest correlation coefficient used as an estimate of the 
reliability of total correct scores on the predictor. 

H. CONSTRUCTION OF RESPONSE PATTERNS 

Since there are 128 (2^) different patterns of responses ranging 
from 0000000 to 1111111, the computer is assigned the otherwise 
tedious and difficult job of constructing and outputting these patterns. 
A difficulty encountered is that leading zeros of various binary 
patterns, although stored without incident in the machine, are lost 
upon printing. Because of this, all zeros in the binary patterns are 
converted to twos. This fact is noted on the printed output (Appendix 
H). 

I. MULTIPLE CORRELATION COEFFICIENT 

The multiple ocrrelation coefficient indicates the strength of 
relationship between one variable and a linear combination of two or 
more others that produces the strongest relationship. Since different 
predictor variables are sometimes interc or related and so duplicate 
one another, the multiple correlation coefficient depends on the 
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intercorrelation of different predictor variables as well as on the 
correlation of each with the criterion variable (Ref. 6). 

Specifically, the multiple correlation between criterion scores 
and a combination of pattern scores and total- correct scores is 
computed . 

Since the coefficient of multiple correlation considers the 
inter-relationship between the predictor variables, it should have, 
theoretically, a greater value than the correlation between either 
predictor and final school grades alone. 

The significance of the multiple r is next computed using an F 
statistic where F is the ratio of the variance of the residuals on the 
criterion before considering the multiple correlation coefficient and 
the variance of the residuals after consideration. 

J. COMPUTATION OF REGRESSION WEIGHTS 

Since there is a possibility that some binary patterns will not 
be used (i. e. , there may be some binary patterns no one has because 
the sample size is small in relation to the number of binary combi- 
nations), it is conceivable that an individual in the cross-validation 
group might have a pattern that no one in the validation gi'oup 
has. Correspondingly, regression weights are computed from the 
relation between total- correct and criterion scores in the validation 
group that are to be used as input in the cross-validation study to 
determine scores for individuals having pattern scores equal to -1. 



24 



K. OUTPUTS I AND II 



The pattern responses are then sorted according to pattern 
score from the lowest (=1) to the highest (73. 66) and, in conjunction 
with the pattern score and total correct of that binary pattern, are 
printed out in tabular form. The table and results thus obtained are 
shown in Appendix H as Output I. 

Next, tables are prepared listing the subject’s in-house identi-. 
fication, his predictor score, his final school grade (criterion score), 
the pattern score associated with his binary pattern, and, finally, 
the total correct scored out of the seven questions. A sample showing 
the first 50 subjects is presented in Appendix I as Output II. 

L. ADDITIONAL OUTPUT 

All correlations and test statistics computed during the 
execution of tlie program are also printed. These results are pre- 
sented and discussed in the RESULTS section. 
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VI. CROSS-VALIDATION 



Cross-validation is a method used to estimate the magnitude 
of sampling variation. In cross-validation, results are obtained 
from a second sample of people for comparison with the results of 
an initial sample. If the results obtained from the second sample 
confirm the results of the first the results are said to hold up under 
cross-validation. 

In addition to the validation or main program, described in the 
preceding section, this study makes use of a cross-validation program, 
which is essentially a portion of the main program. It differs in that 
pattern score and regression weights derived from the previous 
program are read in with new subjects’ personal data and that patterns 
are not constructed, pattern scores are not calculated, and there is 
no need for a joint frequency distribution. The program listing for 
the cross-validation is presented in Appendix J. 

As can be seen from Appendix H, there are fourteen binary 
patterns that were not used by the validation group in the ETST study. 
Therefore, the cross-validation program has to determine if a subject 
has a pattern that was not used in the validation program and, if he 
has, it must assign him a score using the regression weights deter- 
mined from the validation group and his total- correct score. 
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A. ASSIGNMENT OF PATTERN SCORES 



Various other methods of assigning pattern scores to patterns 
that no individual in the validation group has were attempted. These 
methods included: using the average pattern score derived from the 
main program, weighting more heavily those patterns appearing 
more frequently than those appearing less frequently, ignoring a 
subject in the cross-validation who had a pattern no one had in the 
validation group (with adjustment of corresponding variables, e. g. , 
sample size), and finally using the regression weights. 

With only one exception, that of using the regression weights, 
all methods of attack failed. All pattern-score validities were 
significantly lowered in all the other cases. (The reason for the 
abrupt drop in pattern- score correlation coefficients in the cross- 
validation is discussed in the RESULTS section! ) 

Using the regression weights, however, pattern score validi- 
ties maintained a maximum, Scores were obtained by adding the 
product of total correct and the slope regression weight to the 
regressed mean. 

Inputs for the cross-validation consisted of the same information 
as noted in the main program plus the regression weights and the 
pattern scores from the main program. 

The tabular results for the first fifty subjects (even numbers 
only) is presented in Appendix K as Output III. 
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VII. METHOD 



The validation and cross-validation programs were first used 
on GOT data. Not only was the GCT data analyzed, but it also 
served, at the beginning of the research effort, as a test platform for 
debugging the validation and cross-validation computer programs. 

The study concentrated on the ETST data, however. 

A, PRELIMINARY STUDIES 

GCT data were used in prelinriinary studies. Use of GCT data 
as a predictor, as originally planned, was unsatisfactory because 
the GCT was not designed as a predictor of success in a training 
school and, of the seven questions considered in the study, approxi- 
mately one- third of the sample subjects had all correct, which is 
hardly an indication of predictive validity. 

The first step in the study was the determination of the sample 
size to be used in the validation and cross- valididation programs. 
Since the total number of possible combinations of ones and zeros 
was 128 (2 ), it was decided that an appropriate sample size in the 
main program would be 2,000. This would result in the theoretical 
utilization of 15-16 subjects per binary pattern: 

128 patterns “ 
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The remaining subjects in the sample (379) would then be used in 
cross-validation studies. 

Table II summarizes the results of this first effort. As can be 
seen from these results, the greatest validity for the validation 
(main) group was obtained from the predictor vs. criterion scores 
(r = 0. 51). However, the relationship between die pattern and 
criterion scores was only 0. 44. Although lower than the predictor 
validity coefficient, it was still better than the r for total correct 
(0. 37). The high absolute values of the test statistics indicate that 
all the differences were significant. 

The cross validation tells essentially the same story. The 
validities for the pattern and total correct were very 
nearly the same as in the validation program. However, the validity 
for the patterns fell short of its counterpart in the validation program 
(r^val “ • ryal = .44). This phenomenon resulted from the 

weighting of item responses which maximized the correlation with 
the criterion, i.e. , minimized the error of prediction, thereby 
capitalizing on chance in the validation group. The fact that chance 
played an important role in the validation program was further 
illustrated by one subject who had a binary pattern with four ones, 
i. e. , four out of seven correct, but who also had the highest of all 
pattern scores. 

Another explanation for the substantial reduction in pattern 
vs. criterion validities is that the mean pattern score was used for 
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TABLE II 



CORRELATION COEFFICIENTS AND 
TEST STATISTICS DERIVED FROM 
THE GENERAL CLASSIFICATION 
TEST 





Main 


Cross 

Validation 


r(pattern) 


0.44 


0.34 


r(predictor) 


0.51 


0.52 


Z 


-2.72 


-3.11 


r (total ones) 


0. 37 


0.36 


Z 


2.74 


-.27 



NOTE: 1. The first Z is for the difference between the pattern- 

criterion and the predictor- criterion correlations. The second Z 
is for the difference between pattern- criterion and total correct- 
criterion correlations. 

2. The sample size in the main study was 2, 000 subjects 
while the sample size in the cross-validation study was 379 subjects. 
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individuals in the cross-validation group who had patterns no one had 
in the validation group. 

B. ASSIGNMENT OF PATTERN SCORES 

Because of the discrepancies in pattern- score validities for 
both the validation and cross-validation programs, the problem of 
assigning a valid pattern score to an individual who, in the cross- 
validation process, had a pattern no one had in the main validation 
arose. Therefore, the several approaches mentioned earlier were 
formulated and attempted. 

1. The First Solution 

The first of these proposed solutions involved the use of 
weights proportional to the number of subjects having a pattern. 

The weights were to be calculated, along with the pattern scores, 
in the main program and outputted on punched cards. The theory 
behind this solution was that if a binary pattern appeared very 
frequently it should have been counted more heavily in the cross- 
validation than those patterns appearing less frequently. Once again, 
considering the subject who had the highest pattern score with only 
four correct, it would appear logical that that person was not typical 
and should not have been counted equally as others. That is, would 
it have been valid to give his score the same weight as a score 
that was 25 per cent more prevalent? If both scores receive equal 
weight, distortion of the validities must certainly occur. Unfortunately, 
a suitable method of computing and applying such weights was not found. 
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2. The Second Solution 



The second solution was to reduce the number of questions 
used in tlie study from seven to six. With only six questions the 
number of binary combinations would have been significantly reduced 
(from 128 to 64) resulting in the utilization of more binary patterns. 

It was hoped, in fact, that all binary patterns would have been used. 
Thus, when going into the cross-validation phase all patterns would 
have had pattern scores and the need for generating pa ttern- score 
substitutes in the cross-validation could have been eliminated. 
However, even with consideration of only six questions (64 combina- 
tions of ones and zeros), eight binary patterns were not used. 
Furthermore, the magnitudes of the correlation coefficients decreased 
markedly. Therefore, this approach was eliminated. 

3. The Third Solution 

The third solution called for the elimination of those 
subjects in the cross-validation who had a binary pattern no one 
had in the validation smdy. The theory behind this solution was, in 
essence, to eliminate the problem by pretending it wasn't there. 

This solution was not suitable for apparent reasons. For a test to 
be valid in a real environment, vis a vis a laboratory environment, 
it must consider all contingencies. 

4. The Final Solution 

It was finally decided to calculate regression weights and 
use these in assigning pattern scores to subjects in the cross-valida- 
tion who had patterns no one had in the main program. 



32 



C. READ IN OF ALTERNATE DATA 

The possibility of sample bias was also considered, e. g. , 
predictor or criterion scores of the entire sample could have been 
placed in order of increasing or decreasing magnitude. Therefore, 
it was decided to split the sample in half; the first half to be used 
in the validation program and the second half in the cross-validation 
program. The main or validation program was then designed to 
read the records of every alternate subject, e. g. , every odd- 
numbered subject, and make appropriate calculations from those 

data. The cross-validation also read every alternate 
but complementary record. Thus, for example, if the main program 
read every odd record, the cross-validation program correspondingly 
read every even record. Unfortunately, however, splitting the 
sample this way resulted in a drop in the number of 

subjects per binary pattern from fifteen to approximately nine. 

D. THE ETST STUDY 

After solving the problem of assigning pattern scores to 
subjects in the cross-validation study, the research focused on 
utilization of the ETST as the predictor. 

Data preparation followed the same procedures as those 
noted in the data-preparation section of this thesis. 

In addition to the tables and correlation coefficients computed 
in the validation and cross-validation processes, the programs also 
outputted the sum of total correct and the sum of the squares of 
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total correct for all subjects. This was used in the computation of 
the mean and variance for total correct (total ones). The reason 

for these calculations was to determine the test- retest correlation 

coefficient. 

1. Test- Retest Reliability Coefficient 

The test- retest reliability coefficient, as described by 
Weitzman (Ref. 7), is an estimate of the correlation between identical 
versions of a test taken by the same persons in independent trials. 

For a test with n-items and a- alternatives, this estimate is: 

rtt = 1 " n- M 

a^ (1) 

where M and S are the mean and standard deviation, respectively. 

This estimate of the test- retest reliability coefficient 
can be used in tlie determination of the correction for attenuation. 

2. Correction for Attenuation 

Because correlation results are obtained from fallible 
measurements, errors tend to reduce or attenuate the correlation 
between traits. Using the formula for correction for attenuation, it 
is possible to estimate what the correlation would be if perfect, 
errorless measurements were available (Ref. 8). Correlation 
coefficients that are corrected for attenuation cannot be used in pre- 
diction equations but can be used when analyzing relationships to 
make allowances for random errors of measurement. 

Using the test- retest correlation coefficient computed for 
the predictor, it is possible to calculate the validity of the predictor 
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corrected for attenuation. The value obtained from the following 
formula is the theoretical correlation coefficient if the predictor 
were error- free: 

r r = 

^00 C j= , . 

A 1 pp (2) 

The correlation coefficient rpQ is the measured validity between 
the predictor and criterion, and i*pp is the test- retest reliability 
described in the previous section. A comparison between the 
validity coefficient (rp^) and the validity coefficient corrected for 
attenuation (r^Q, q) was used as an indication of how close this study 
came to the theoretical limit of validity for the predictor. Specifi- 
cally, r (total correct vs. final school grade) was compared to 
the corresponding correlation coefficient corrected for attenuation. 
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VIII. RESULTS 



A. DETERMINATION OF LINEARITY 

A product- moment correlation coefficient is good only if a 
linear relationship exists between the variables that are being corre 
lated. Figures 2 and 3 are scatter diagrams which were used to 
determine if a linear relationship existed between total correct out 
of seven and final school grade and the full ETST score and the 
final school grade. Note that almost all the points can be enclosed 
in an oval which goes from the lower left to the upper right, there- 
fore indicating linearity (Ref. 9). 

B. CORRELATION COEFFICIENTS AND TEST STATISTICS 
Table III lists the values for all test statistics and correlation 

coefficients. 

As can be seen from that table, the value of fpattern score 

decreases from 0. 76 in the validation program to 0. 72 in the 
cross-validation program, the reduction due to maximization of 
chance in the main program. The computation of the multiple 
correlation coefficient was desired to see if pattern scores add to 
the predictive ability of total- correct scores. The 
multiple correlation coefficient did not increase the value of 
^total correct indicating no additional predictive ability. The 
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FINAL SCHOOL GRADE (CRITERION) 



FIGURE 3 
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TABLE III 



CORRELATION COEFFICIENTS AND 
TEST STATISTICS DERIVED FROM 
THE ELECTRONICS TECHNICIAN 
SELECTION TEST 





Main 


Cross- 

Validation 


r(pattern score) 


0.76 


0.72 


r(predictor) 


0. 61 


0.60 


Z 


7.13 


4.93 


r(total correct) 


0.72 


0.73 


Z 


2. 26 


-.54 


r(pattern score/ 






total correct) 


0.95 


0.95 


r(multiple) 


0.76 


0.73 


F 


176.36' 


15.38 


r(test- retest) 




0.73 


Correction for 






Attenuation 




0. 85 



NOTE: 1. The first Z is for the difference between the pattern- 

criterion and the predictor- criterion correlations. The second Z 
is for the difference between pattern- criterion and total correct- 
criterion correlations. 

2. The sample sizes in both the main and cross-validation 
studies was 1,182 subjects. 
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large value of F indicates that the total- correct scores contributed 
significantly to the predictive ability of the pattern scores, 
however. 

The high value of the correlation coefficient between pattern 
scores and total- correct scores indicated that the seven items used 
in the study constituted a very valid test and that the total correct 
could be used as a predictor that is as good as the pattern scores 
for these items. 

The correction for attenuation revealed that the highest validity 
theoretically obtainable by improving the reliability of the seven- 
item predictor was 0. 85, The value actually obtained, 0. 73, was 
equal to the test- retest reliability of the test. Since it is not reason- 
able to expect that a test will correlate more highly with another 
test than it does with itself, it is no wonder that the pattern scores 
did not correlate better with the criterion than the total- correct did. 
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IX. CONCLUSION 



The two FORTRAN computer programs developed in this 
study successfully determined and correlated pattern scores with 
the criterion. However, the questions extracted from the ETST 
were so highly valid that they could have been used alone, i, e. , 
without pattern scoring, as predictors of success in the Basic 
Electronics and Electricity School. 

It would be interesting to continue this smdy using biographical 
information, not ordinarily quantifiable, instead of extracts from 
current examinations. BiographicM questions carefully constructed 
and easily verifiable could be used in predicting behavior, and pattern 
scoring is a method that can be used to quantify responses to these 
questions. Responses quantified by pattern scoring, in fact, will 
show the highest possible correlations with predicted behavior. 
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APPENDIX A 



The Seven Best GCT Items Selected by SEQUIN 



^T) 
Form 7 
Item 
Number 


(2) 

Item 

Type 


Recruit 
p Value 


(4) 

Median 
School 
p Value 


(5) 

Median 

School 

Validity 


13 


A 


.77 


.88 


.22 


19 


SC 


.60 


.78 


.20 


31 


A 


.75 


.85 


.20 


55 


A 


.41 


.49 


. 24 


62 


SC 


.60 


.69 


,26 


67 


SC 


.80 


.87 


.26 


94 


SC 


.55 


.78 


.30 



NOTE: 1. Values in Columns (4) and (5) are based on item data 

only for schools in which that item was selected in Program SEQUIN, 
2. A = analogies; SC = sentence completion item. 
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APPENDIX B 



The Seven Best ETST Items Selected by SEQUIN 



Question 

Number 


Item 

Type* 


Recruit 
P- Value 


Median School 
P- Value 


Median 

School Validity 


3 


M 


.57 


.71 


.34 


11 


M 


CO 

CO 


.58 


.44 


13 


M 


.58 


.69 


.32 


22 


S 


.57 


.77 


CO 


40 


S 


.21 


CO 


.40 


41 


E 


.31 


.39 


.26 


50 


E 


.25 


.31 


OO 



* M = Math; S = Science; E = Electricity or Radio 
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APPENDIX -C 



FIRST DATA PREPARATION PROGRAM 



READ CARDS 

1 . 2 , 5,6 



V 

ll\f - STAR, BMNK, dash) 



YES 




NO 



V 
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APPENDIX D 



Listing of First Data Preparation Program 
C 
C 

C THIS PROGRAM EDITS RAW DATA FOR USE IN ETST STUDY 
C 

INTEGER*2 CASH, ZERO, BLANK, IW, 1C3, IC4, IC5 
DIMENSION IW(80) 

DATA DASH/'- '/,ZERO/'0 '/, BLANK/' '/,IC3/'3 '/,IC4/ 
lIC5/'5 '/,IC6/'6 '/,STAR/'^ '/,ICl/'l '/,IC2/'2 '/ 

K=1 

C 

C 

C CHECK CARD NUMBER 
10 READ(4,4C0,END=500) IW 

IF( IW(8 ) .EC. ICl ) GO TO 12 
IF( IW( 3) .EO. IC2) GO TO 12 
IF( IW(8).E0.IC3) GO TO 10 
IF( IW(8) .E0.IC4) GO TO 10 
IF( IW( 8) .E0.IC5) GO TO 12 
IF( IW{8) .E0.IC6) GO TO 12 
400 FORMAT(80A1) 

C 

C 

C ZEROIZE STARS, BLANKS, CASHES 



12 


DO 20 1=1,80 






IF( IW( I ) .EO.ST AR) 


IW! I )=ZERO 




IF( IW( I ) .EO. BLANK) 


IW'!I ) =ZERO 




IF( IW( I ) .EO.DASH ) 


IW! I ) =ZERO 


20 


CCNT INUE 
WRITE! 8,300) IW 
WRITE(6,401) IW 




300 


FORMAT (80A1) 




401 


FORMAT! 1X,80A1 ) 
GO TO 10 




500 


IF!K.GE.2) GO TO 99 

K=K+1 

GO TO 10 


99 


STOP 

END 





//GO.FT06F001 DD S PACE= ( CYL , ( 5 , 5 ) , RLS E ) 

//GO.FT J4F001 DD UNI T=2400 , VOL=S E R=NPS4 1 6 , D I S P = ( OL D, PASS ) 
// DCB=(RECFM=FB,LRECL=8C,BLKSIZE=4800) , LA BE L = ( 1 , NL , , I N ) , 
// DSN=£F1 

//GO.FT04F002 DD UNI T=2400 , VOL =SER=NPS41 6 , D I S P = ( CL D , P AS S ) 
// DCB=(RECFH=FB,LRECL=80,BLKSIZE=4800) ,LABEL=( 2,NL, ,IN) , 
// DSN=£F2 

//G0.FT08F001 DD D I S P= (N E W ,KEEP ) , UNI T =232 1 , VO L = S EP =C ELO 01 , 
// LABEL=EXPDT = 73180,SPACE=(TRK, (571) ) , D SNAME = S C575. KPW2 , 

// OCB=(RECFM=FB,BLKSIZE=2000,LRECL=80 ) 



APPENDIX E 



SECOND DATA PREPARATION PROGRAM 



"^AD 1 

5.- 


CARDS 

6 


\ 


/ 



COMPUTE RANGE 
OF CRITERION SCORES 
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c 

c 

c 

c 

c 

c 



c 

c 

c 



c 

c 



c 

c 

c 



APPENDIX F 

Listing of Second Data Prepation Program 

THIS PROGRAM EDITS DATA FROM THE ETST TEST 
A( J)=PREDICTOR, C(J) =CRITERIONtD(J )=SERV.NO. , 

IMPLICIT I\'TEGER=^^4( A-Z ) 

DIMENSION A(2500), C(2500) , D(2500 )tW(7) 

DATA ISAMP,NREAD,NWRITE,NPUNCH, I HI » I LO/240 0 ,8 ,6 ,7 , 
156,56/ 

DO 100 J=1,ISAMP 

IF{ J.EO. 2398) GO TO 50 

10 REAOINREAD, 1, ENQ=50) B , D( J ) , KDNUM , ( W ( I ) , I = 1 , 7 ) 

1 FORMAT (Al, 16,1 1 ,T11 , II ,T19, 1 1 ,T21 , 1 1 ,T3 0, I 1,T48, I 1, 
1T49, I 1,T58,I 1) 

THIS PHASE CHECKS FOR THE PRESENCE Of CARD NUMBERS 
FIVE AND SIX 

IF(KDNUM.EQ.5J GO TO 3 
GO TO 10 
K=1 

READ(NREAD,2 ) K DNUM , A ( J ) , C ( J ) 

FORMAT (T 8, II ,T 52 , 12 ,T64, 12 ) 

IF(KDNUM.E0«6) GO TO 7 
GO TO 10 
K = K+1 

IF(K.NE.2) GO TO 10 

THIS PHASE DETERMINES RANGE OF CRITERION SCOFES 
IF(C(J) .EO.O) GO TO 12 
IF(C(J) .LT.ILO) ILO=C(J) 

IF(C( J) .GT.IHI ) IHI=C(J) 

THIS PHASE DETERMINES CORRECT / I NCORR ECT RESPONSES 
OF SELECTED EJ-ST ITEMS NOS. 3,11,13,22,40,41, AND 50 
12 IF(W(1) .NE.5) Wm = 0 

IF(W(2 ) ,NE.5) W(2)=0 
IF( W( 3) .NE. 5) W(3)=0 
IF(W(4) .NE.2) W(4)=0 
IF(W(5) .NE.3) V')(5)=0 
IF(W(6) .NE.2) W(6)=C 
IF(W(7) .NE.3 ) W(7)=0 
DO 20 1=1,7 

IF(W( I ) . EO.O) GO TO 20 
W( I ) =1 

20 CONTINUE 



OUTPUT INFO ONTO CARDS, DATA CELL, PAPER 
•J* IS USED AS INHOUSE ID 

WRITE{NWRITE,30) (W(I),I=1,7),C(J),A(J) ,J,B,D(JJ 
FORMAT (4 (IX i7 1 1 , 1 2 , 1 2 , 14 , T 20 , A 1 , I 6, 5X ) ) 

WRITE( NPUNCH ,40) (W(I),I=l,7),C(J),A(J),J,e,D(J) 

WRITE(4,40) (W(I),I=1,7),C(J),A(J),J,B,D(J) 

FORMAT! 711 , 12, 12 , 14 , T2 0 , Al , 16 ) 

CONT INUE 

WRITE(6,25) IHI,ILO 

FORMAT!' IHI EQUALS ' , I 2 , / ' ILO E QU ALS ' , I 2 , / ) 

STOP 

END 

//GO.FT06F001 DD SPA CE= ! CYL , ! 5 , 5 ) , RLS E ) 

//G0.FTC7FG01 OD SYSOUT=B 

//GG.FT08F001 DD DSN = S05 7 5 . KPW2 , UN I T= 232 1 , VOL = SE R=C EL 001 , 

// DCB = !RECFM=FB,BLKSI ZE=2 000 , LRECL=80 ) , D I S P= ! CL D, K E E-? ) 

//G0.FT04F001 DO D IS P = ! NE W , KEE P > , UNIT =23 21 ,-VOLUME = SER=CEL001 
// LABEL=EXPDT=73180,SPACE=!TRK, (571) ) , DSNAM E= S0575 . KPW3 , 
// DCB = !RECFM=FB,B LKSI ZE=2 000, LRECL = 80 ) 



30 



40 

100 

50 

25 
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APPENDIX G 

Listing of Validation Program 

THIS PROGRAM WORKS ON ODD NUMBERED QUESTIONS FROM THE 
ETST EXAM. THE MULTIPLE CORRELATION COEFFICIENT IS 
CALCULATED AS WELL AS THE CORRELATION COEFFICIENT 
BETWEEN PATTERN AND T OTAL CORRECT. 



INTEGER*4 A »C t D, E , G ,H , P 

REAL=^'8 C1»C2,A1 ,A2,V,W,X,R1,R2, R3, R4,R5, Q, Z1,Z2,Z3,Z, 
IRlSTARtAA.BB 

DIMENSION A(1200)»B(1200),C( 12 00 ) , D ( 1200 ) ,E(1200) , 
lF(128t47),G(123) » H ( 128 ) , P { 7 , 1200 ) , S ( 128 ) , X { 1200 ) 

DATA N1»N2,N3»N4/1182,7,47»128/ 

L=0 



READ IN DATA 
DO 13 J=ltNl 

READ(9.9) { P( I, J )»I=1»N2),C{J)»A{J),D(J) 
FORMAT (711 , I2» I2» 14) 



IDUM IS A DUMMY VARIABLE CONTROLLING THE READING OF 
EITHER EVEN OR ODD DATA. 

READ(9,2) IDUM 
2 FORMAT! ID 
13 CONTINUE 



COUNT TOTAL ONES FOR EACH SUBJECT 

5 DO 15 J=1»N1 
E( J)=0 

DO 12 1=1, N2 
E( J) =P{ I ,J)+E( J) 

12 CONTINUE 
15 CONTINUE 
IA1 = 0 
IA2=0 
IV=0 

DO 8 J=1,N1 



lAl, IA2, IV ARE VARIABLES TO BE USED IN THE CALCU- 
LATION OF RITOTAL ONES) LATER IN THE PROGRAM. 

IA1=E( J)+I A1 
IA2=E{ J)*E(J)+IA2 
IV=C( J )*E( J )+IV 
8 CONTINUE 



OUTPUT VALUES FOR I Al , IA2 TO BE USED IN THE 
COMPUTATION OF TEST-RETEST CORRELATION COEFFICIENT. 

WRITE(6,999) IA1,IA2 

FORMAT (T20, ' I A 1= • , I 8 , / / , T2 0 , • I A2 = ' , I 8 ) 



DETERMINE THE JOINT FREGUE1NCY DISTRIBUTION OF PATTERN 
AND CRITERION SCORES (THE SECOND I LOOP CONVERTS 
BINARY NUMBER PATTERNS TO DECIMAL EQUIVALENTS 
TO SERVE AS ROW ADDRESSES. 

DO 14 I=1,N4 
DO 17 J=1,N3 
F{ I, J)=0 
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Listing of Validation Program 
(Continued) 



17 CCNTINUE 
14 CONTINUE 

CO 18 J=1»N1 

M=1 

K=N4 

DO 19 I=1»N2 
K=K/2 

M=K*P{ I , J )+M 
19 CCNTINUE 
N=C( J)-29 
F(M,N)=F(M,N)+1 
B{ J)=M 

18 CONTINUE 



CCMPUTATION OF PATTERN SCORES 

CO 20 1=1, N4 
S1 = 0 
S2=0 

DO 21 J=1,N3 
S2=F(I , J)+S2 
Sl=( J+29)*F( I, J)+S1 
21 CONTINUE 

IF(S2.F0.0) GO TO 10 
S( I ) = S1/S2 
GO TO 20 
10 S(I)=-1 

20 CONTINUE 

WRITE (7, 25) (S( I) ,T=1,N4) 

25 FORMAT! 10F7-4) 



ASSIGNMENT OF PATTERN SCORES TO SUBJECTS 

CO 31 J=1,N1 
K=B{ J) 

X{ J ) = S(K ) 

31 CCNTINUE 



COMPUTATION OF CORRELATIONS 

A1=0.D0 

A2=0.D0 

C1=0.D0 

C2=0. DO 

X2=0.00 

X1=0.D0 

V=0.D0 

W=0.D0 

UU=0. 

DO 41 J=1,N1 
C1=C( J)+C1 
C2=C( J )*C(J )+C2 
A1=A (J) +A1 
A2=A(J )*A( J)+A2 
X1=X( J ) +X1 
X2=X( J)=5=X( J)+X2 
V=C(J )=^A( J )+V 
W=C{ J)^ X(J )+W 
UU=E{ J )v-X{ J)+UU 
41 CONTINUE 

R3={N1*C2)-(C1*C1) 

R2 = (Nl=j'X2)-( X1*X1) 

R5 = (N1*W )-{Cl^Xl) 

0={R2=^R3)*=S=0.5 

R2=R5/0 



FIRST TIME THROUGH PROGRAM R1 IS THE CORRELATION 
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Listing of Validation Program 
(Continued) 



( 



COEFFICIENT FOR PATTERN SCORE VS. CRITERION. SECOND 
TIME THROUGH R1 IS CORRELATION COEFFICIENT FOR TOTAL 
ONES VS. CRITERION. 

30 Rl = INl>l'A2) -IA1=^A1 ) 

R4=(N1#V)-IC1'-;'A1 ) 

0= IR1=5<R3 
Rl=R4/0 

IF(L.EO.l) R1STAR=R1 



COMPUTATION OF TEST STATISTIC FOR R DIFFERENCE 

Z1={1+R1)/ (1-Rl ) 

Z1=DL0G ( Zl) /2 
IF(L.EO.l) GO TO 40 
Z2=(l+R2)/ (1-R2) 

Z2=DL0G(Z2)/2 
Z3=2./(Nl-3. ) 

Z3 = (Z3)^==S'0. 5 
Z=(Z2-Z1 )/Z3 
IF (L.EO.l) GO TO 90 



CCNSTRUCTION OF RESPONSE PATTERNS 

GI 1 ) = 2 
H(1)=0 
GI 2)=1 
H(2) = l 
K = 1 
N=1 

50 N=2*N 

IFIN.GE.N4) GO TO 60 
K=10*K 
DO 51 1 = 1. N 
G(N+I)=Gn )+K 
H(N+I)=H(I )+l 
GI I )=GI I )+2>’'R 

51 CCKTINUE 
GO TO 50 



ORDERING OF RESPONSE PATTERNS 

60 N=N4-1 

70 K = 0 

DO 80 1=1. N 
V=GI I ) 

W=HI I ) 

IF ISI I+l) .GE.SI I ) ) GO TO 80 
U=SII ) 

V=GII) 

. W=HI I ) 

SI I)=S I I+l ) 

GI I)=GI I+l) 

HI I)=HI I+l) 

SI I+l ) =U 
GI I+l) =V 
HII+1)=W 
K = 1 

80 CONTINUE 

IF IK.EO.l) GO TO 70 



PRINT OUT 

NUM = 1 
NUM1=32 

99 WRITEI6, 100) IGI I ) , H I I ) , S I I ) , I = NUM,NUM1) 

100 FGRMATI'l* .61/) ,T60,'ETST EX AM ' , 2 I / ) ,T45 , • + • . 
838I'-').'+',/.T45,'l'.lX,' PATTERN* , 



50 



ooonnno oonnn nonnn 



Listing of Validation Program 
(Continued) 



THE VARIABLE R7 IS THE CORRELATION COEFFICIENT BETWEEN 
PATTERN SCORES AND TOTAL CORRECT (ONES). 

R7=N1=^<UU-IA1*X1 
WRITE(6,33) R7.UU 

33 FORMAT!' R7= » , F i 8 . A , 'UU= ' , F 1 8. 4 ) 

37 FCRMATC R7 = » , F18. 4 , »UU= ' , F18 .4 ) 

Rl=( N1=^A2)-(A1*A1) 

R8=(N1':-=X2)-(X1*X1) 

0=(R1*R8 

R7=R7/0 

WRTTE(6,213) R7 

213 FORMATCO' ,'R(PATTERN SCORE/TOTAL ONES) EQU ALS ' , F6. 3) 



RMUL IS THE MULTIPLE CORRELATION COEFFICIENT TO BE 
USED IN THE DETERMINATION OF »F’. 

RMUL = ( (R2^*2) + (R1STAR^*2)-2*(R2’^=R1STAR=<=R7) )/ ( l-R7=5-'*2 ) 
RMUL=RMUL**.5 

FF=((RMUL^*2)-(R1STAR^*2) )=?(Nl-3)/( l-( RMUL*=!'2) ) 
WRITE(6,417) RMUL.FF 

7 FORMAT! ‘O' , «R( MULT. CORREL. CQEF. ) E DUAL S ' ,4X , F6 .4 , // , 
1» FF EQUALS' ,F8.4) 



THIS PORTION OF THE PROGRAM IS USED IN THE DETERMINA- 
TION OF A FREQUENCY DI STRI BUT I ONF I.E. PATTERN 
VS. NUMBER OF PEOPLE WITH THAT PATTERN 



DO 350 1=1, 128 
WR1TE!6.357) G!I),S!I) 

357 F0RMAT!T20,'A PATTERN OF : ' , 2X , I 7 , 2X , • AND PATTERN SCORE 
13X,F8.4, //) 

KGUMT=0 
DO 355 J=1 ,N1 

IF!S! I ) . EO.X! J ) ) GO TO 359 
GO TO 355 

359 K0UNT=K0UNT+1 

WRITE!6,352) D!J) 

352 F0RMAT!T15,I4) 

355 CONTINUE 

WRITE(6,351) KOUNT 

351 FORMAT!' TOTAL PEOPLE HAVING THIS S CORE : ' , 1 3 , // ) 

350 CONTINUE 
STOP 
END 

//G0.FT06F001 DD SPACE = ! C YL , ! 5 ♦ 1 ) ) , S YSOUT= D 
//GO. FT07F001 DD SYS0UT=6 

//GO. FT09F001 DD DSN=S 057 5 . K PW3 , UN IT =232 1 , VOL =S ER=CELOD 1, 

// OCB=!RECFM=FB,6LKSIZE = 2 000,LRECL = 80) ,DI SP= (OLD, KEEP) 
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APPENDIX H 



NOTE 2: 



Output I - Pattern Information 



ETST EXAM 



PATTERN 


TOTAL ONES 1 


1 PATTERN SCORE 


2222112 


2 


-1,0000 


2222111 


3 


-1.0000 


2212121 


3 


-1.0000 


2212111 


4 


-1. 0000 


2122121 


3 


-1.0000 


2122112 


3 


-1.0000 


2122111 


4 


-1 . 0000 


2121122 


3 


-1.0000 


2112211 


4 


-1.0000 


2112112 


4 


-1.0000 


1212111 


5 


-1.0000 


1122121 


4 


-1.0000 


1122112 


4 


-1.0000 


1122111 


5 


-1.0000 


2212211 


3 


48.5000 


2222211 


2 


49.0000 


2222222 


0 


49.6071 


1222222 


1 


51.3182 


2222121 


2 


52.0000 


212221 1 


3 


52,0000 


2221222 


1 


52.5238 


2222212 


1 


52.5833 


1222121 


3 


53.5000 


2222221 


1 


53.6667 


2122222 


1 


53.8000 


2121211 


4 


54.0000 


2222122 


1 


54.1667 


2221212 


2 


54.3000 


2212122 


2 


54.50C0 


2212222 


1 


54.5500 


1212222 


2 


54.7500 


1222212 


2 


54. 8889 



NOTE l; IN PATTERNS, 2’SREPR5SENT O' S 
A PATTERN SCORE OF -1 INDICATES A PATTERN MO ONE HAS 
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APPENDIX H 



NOTE 2: 



Output I - Pattern Information 

ETST EXAM 



+ + 



PATTERN 


TOTAL ONES 


1 PATTERN SCORE 


2221221 


2 


56.0000 


2112111 


5 


56.0000 


1222122 


2 


56.3333 


2122221 


2 


56.5000 


2122212 


2 


56.5000 


1222112 


3 


56.5000 


1212122 


3 


56.8000 


2121212 


3 


56.8750 


2212212 


2 


57.2500 


1122222 


2 


57.3333 


1221222 


2 


57.3913 


1122221 


3 


57.6000 


2211222 


2 


57.7308 


2221121 


3 


58.0000 


2121222 


2 


58.0000 


2112212 


3 


58.0000 


2211212 


3 


58.2941 


1212212 


3 


58.3077 


1221221 


3 


58.3333 


2221211 


3 


58.5000 


2211122 


3 


58.7143 


1212221 


3 


58.8750 


2211221 


3 


58.9091 


2221122 


2 


59.0000 


2211211 


4 


59.0000 


1222111 


4 


59.0000 


1221212 


3 


59.2222 


1221211 


4 


59.2500 


1222221 


2 


59.5000 


2121112 


4 


59.6667 


2112222 


2 


59.7143 


1211222 


3 


60.0682 



NOTE l; IN PATTERNS, 2'SREPRESENT O’S 
A PATTERN SCORE OF -1 INCICATES A PATTERN NO ONE HAS 



(Continued) 
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APPENDIX H 



NOTE 2: 



Output I - Pattern Information 



ETST EXAM 



PATTERN 


TOTAL ONES 1 


1 PATTERN SCORE 


2111222 


3 


60.7333 


1221122 


3 


60.7500 


2121221 


3 


61 . 0000 


1121222 


3 


61.0000 


1112122 


4 


61.2500 


2212221 


2 


61.6667 


1212211 


4 


61.6667 


2221112 


3 


62.0000 


2212112 


3 


62.0000 


2122122 


2 


62.0000 


2111212 


4 


62.2222 


1211221 


4 


62.2308 


2112121 


4 


62.5000 


1212112 


4 


62.5000 


1122212 


3 


62.5714 


1112222 


3 


62.8621 


2112221 


3 


63. OOCO 


1112212 


4 


63.2609 


1211212 


4 


63.2857 


1122122 


3 


63.5000 


1121212 


4 


63.5625 


1211122 


4 


63.6000 


1112121 


5 


63.7500 


2121121 


4 


64.0000 


2112122 


3 


64. 0000 


1211211 


5 


64.0000 


1122211 


4 


64.2500 


1111222 


4 


64.4000 


2111122 


4 


64.6000 


1121122 


4 


64.8000 


2211121 


4 


65.0000 


2111221 


4 


65. OOCO 



NOTE i: IN PATTERNS^ 2'SREPRESENT O’ S 
A PATTERN SCORE OF -1 INDICATES A PATTERN NO ONE HAS 



(Continued) 
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NOTE 2: 



Output I - Pattern Information 

ETST EXAM 



+ + 



PATTERN 


TOTAL ONES 


1 PATTERN SCORE 


2211112 


4 


65.2000 


1221112 


4 


65.2000 


1211112 


5 


65.300J 


1121221 


4 


65.3000 


1112211 


5 


65.5000 


1111212 


5 


65.6097 


1112112 


5 


65.7500 


2111121 


5 


65 .7778 


1112221 


4 


65.8571 


2111112 


5 


66.0000 


1222211 


3 


66.0000 


1121112 


5 


66.3333 


2121111 


5 


66,5000 


2111211 


5 


66.7500 


1111221 


5 


66.7667 


1111122 


5 


67.7273 


2221111 


4 


68 .0000 


1221111 


5 


68.3333 


1211121 


5 


68.5000 


1121211 


5 


68.5000 


1111211 


6 


68.5000 


1111112 


6 


68.7500 


1212121 


4 


69.5000 


1121121 


5 


69.6667 

. . . 


2211111 


5 


69.8333 


2111111 


6 


70.0000 


1112111 


6 


70.5000 


1211111 


6 


70.8000 


1221121 


4 


71 .0000 


1111121 


6 


71.2954 


1111111 


7 


73.2857 


1121111 


6 


73.6667 



NOTE l: IN PATTERNSt 2'SREPRESENT O'S 
A PATTERN SCORE OF -1 INDICATES A PATTERN NO ONE HAS 



(Continued) 
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Output II - Subject Information 

ETST EXAM 



I DENT 


PREDICTOR 


CRITERION 


1 PATTERN SCORE 


1 TOTAL ONES 


1 


66 


76 


68.7500 


6 


3 


59 


53 


54. 7500 


2 


5 


57 


54 


54.7500 


2 


7 


73 


73 


71.2954 


6 


9 


56 


60 


58. 9091 


3 


11 


71 


70 


66.0000 


5 


13 


63 


64 


64.4000 


4 


15 


59 


57 


54.5500 


1 


17 


64 


56 


57.3913 


2 


19 


64 


54 


49.6071 


0 


21 


46 1 


58 


54.7500 


2 


23 


56 


56 


58.8750 


3 


25 


56 


60 


65. 6097 


5 


27 


55 


54 


58.2941 


3 


29 


65 


68 


68.7500 


6 


31 


49 


57 


54.7500 


2 


33 


55 


62 


60. 0682 


3 


35 


72 


76 


73.2857 


7 


37 


68 


71 


68.7500 


6 


39 


66 


68 


65. 8571 


4 


41 


73 


71 


70. 0000 


6 


43 


65 


60 


64.4000 


4 


45 


59 


59 


61.0000 


3 


47 


63 


76 


73.6667 


6 


49 


65 


64 


62.5000 


4 


51 


54 


38 


51.3182 


1 


53 


66 


64 


62.8621 


3 


55 


58 


62 


57.2500 


2 


57 


51 


59 


57.2500 


2 


59 1 


61 


64 


65.2000 


4 


61 


62 


68 


65.3000 


4 


63 


63 


67 


70.0000 


6 


65 


60 


62 


56.0000 


2 


67 


66 


70 


68.5000 


5 


69 


68 


69 


63.6000 


4 


71 


6Q 


68 


68. 5000 


6 


73 


62 • 


60 


54.5500 


1 


75 


58 


60 


56. 8750 


3 


77 


65 


59 


54.7500 


2 


79- 


58 


58 


61.0000 


3 


81 


62 


57 


56.0000 


2 


83 


63 


71 


69.5000 


4 


85 


59 


58 


57.7308 


2 


87 


70 


76 


73.2857 


7 


89 


59 


70 


65.7778 


5 


91 


51 


61 


60. 0682 


3 


93 


51 


49 


52. 5238 


1 


95 


45 


60 


57.6000 


3 


97 


66 


68 


68. 5000 


5 


99 


71 


65 


68.7500 


6 
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APPENDI X J 



Cross-Validation Listing 
CROSS VALIDATION 
INTEGER*^ A,C» D, E,G,H, P 

REAL=?«8 Cl,C2»AltA2t V,W,X,Rl,R2»R3 5R4,R5tQ,Zl,Z2,Z3,Z, 
1 AA,BB 

DIMENSION A( 1200) ,B (12 00) ,C( 12 00) ,D(1230) ,E(12 00), 

1G( 128) , H( 128) tP( 7t 1200 ) , S ( 1 28 ) , X ( 1200) 

NOTE: PARAMETERS OF DATA CARD 

CATA N1,N2,N3, N-^/1182, 7,47, 128/ 

L = 0 



READ IN CATA 
DO 13 J=1,N1 

THIS CROSS VALIDATION PROGRAM READS DATA FROM THE 
ETST EXAM 



IDUM IS A DUMMY VARIABLE CONTROLLING THE READING OF 
EITHER EVEN OR CCD CATA. 

READ! 4, 2) IDUM 
2 FORMAT (ID 

READ (4,9) (P(I ,J) , 1=1 ,N2) ,C(J ) ,A( J) ,D( J ) 

9 F0RMAT(7I1,I2,12,I4) 

13 CONTINUE 



READ REGRESSED MEAN AND WEIGHT COMPUTED FROM THE 
MAIN PROGRAM. 

READ(5,3) AA,BB 
3 F0RMAT(F6.3,F6.3 ) 



READ PATTERN SCORES CALCULATED FROM MAIN PROGRAM 

READ(5, 10) ( S( I ) ,I = 1,N4) 

10 FORMAT (10F7. 4) 



COUNT TOTAL CORRECT FOR EACH SUBJECT 

DO 15 J=1,N1 
E( J)=0 

DC 12 1=1, N2 
E( J)=P( I ,J)+E( J) 

12 CONTINUE 
15 CONTINUE 
' " IA1=0 

IA2=0 
I V=0 



lAl, IA2, IV ARE VARIABLE TO BE USED IN THE CALCU- 
LATION OF R(TOTAL ONES) LATER IN THE PROGRAM. 

CO 8 J=1,N1 
IA1=E( J)+I A1 
IA2=E(J)’?=E( J) + IA2 
IV=C(J)*E( J)+IV 
8 CONTINUE 
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Cross-Validation Listing 
(Continued) 



OUTPUT VALUES FOR lAl, AND IA2 TO BE USED IN THE 
COMPUTATION OF TEST-RETEST CORRELATION COEFFICIENT. 

WRITE(6,999) IA1,IA2 

999 FORMAT (T20, 'IA1=',I8,//,T20,'IA2=',L8) 



CALCULATE DECIMAL EQUIVALENT OF BINARY PATTERNS 
THE B(J) ARRAY HOLDS DECIMAL EQUIVALENT OF EACH 
SUBJECT'S BINARY PATTERN. 

D0 18 J=1.N1 
M=1 
K = N4 

DO 19 I =1,N2 
K = K/2 

M = K=^^P( I , J)+M 
19 CONTINUE 
B( J)=M 

18 CONTINUE 

ASSIGNMENT OF PATTERN SCORES TO SUBJECTS 
DO 31 J=1,N1 
K=B( J ) 

X( J)=SIK) 

IF( X( J) . LT.O) X{ d) =E( J)=^BB+AA 
31 CONTINUE 

COMPUTATION OF CORRELATIONS 

Al=O.DO 

A2=O.DO 

C1=O.DO 

Xl=O.DO 

X2=0. DO 

W=O.DO 

V=0. DO 

UU=0. 

C2=0.D0 
CO 41 J=1,N1 
C1=C( J)+C1 
C2=C(J )^C( J)+C2 
A1=A( J )+Al 
A2=A( J)*A( J)+A2 
X1=X( J )+Xl 
X2=X( J)*X( J)+X2 
V=C( J)=^^A(J) + V 
W=C(J)*X(J )+W 
UU=E( J)^X( J)+UU 
41 CONTINUE 

R3 = (Nl*C2)-{Cl=i^l ) 

R2=(Nl*X2)-( X1>-X1 ) 

R5=(M1*W)-(C1^X1 ) 

0-=(R2=?R3)'!*0.5 

R2=R5/0 



FIRST TIME THROUGH PROGRAM R1 IS THE CORRELATION 
COEFFICIENT FOR PATTERN SCORE VS. CRITERION. SECOND 
TIME THROUGH R1 IS CORRELATION COEFFICIENT FOR TOTAL 
CORRECT VS. CRITERICN. 

30 Rl = (Nl=5'A2)-( Altf'Al ) 

R4={ N1^V)-IC1^A1 ) 

0=(R1^R3)**0.5 

Rl=R4/0 

IF(L.EO.l) R1STAR=R1 



COMPUTATION OF TEST STATISTIC FOR R DIFFERENCE 
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Cross-Validation Listing 
(Continued) 



Z1=(1+R1 )/ ll-Rl ) 
Z1=DL0G(Z1 )/2 
IF(L.EO.l) GO TO 40 
Z2=(l+R2)/(1-R2 ) 
Z2=DL0G(Z2)/2 
13 = 2 ./ ( Nl-3,) 

Z3 = (Z3)=i^^0.5 
40 Z=(Z2-Z1)/Z3 

IF (L.EO.l) GO TO 90 



PRINT OUT 

NUM2=1 

NUM3=50 

199 WRITE(6,200) (D ( J ) ,MJ),C(J),X(J),E{J), J = NUM2,NUM3) 

200 FORMAT ( • 1» t4(/ ) ,T6 0, ' ETST' ,T67 , ' EXAM* , 2(/) ,T34» ' + ' , 
160( ) , • + • » /,T34, ' I’ t IX, ' IDENT' , IX, • I • , IXt 

2* PREDICTOR* ,1X, * 1 ' , IX , * CRIT ER ION ' , IX, ' I • , 

3* PATTERN SCORE • , 1 X , ' I ' , 1 X , ' TOT AL ONES' , * I * ,/ ,T33 , 
4* I • ,T95, • I • ,/ • + ' ,T35, 6 0( •-•),/, 1 01 5 ( T34 , ‘ I* ,15, 
52X,' I • ,T49 ,I2,T55, ' I ' ,T61, I2,T66, • 1 ' ,T71,F8.4, 

6T82, • I ' ,T89,I1,T95, ' I ' , / ) , T34 , ' 1 * ,T95,' I , 
7T35,60( ),/) ) 



THE NEXT TWO IF STATEMENTS CONTROL THE NUMBER 
AND LENGTH OF THE LAST TABLE 



THE FIRST 'IF* STATEMENT; NUMBER INSIDE PAREN 
SHOULD BE ONE MULTIPLE OF *5* HIGHER THAN N1 ; E.G. IF 
Nl=627, NUMBER INSIDE PAREN SHOULD BE 630; IF Nl=986, 
NUMBER INSIDE PAREN SHOULD BE 990. 

IF(NUM3.E0.1190) GO TO 221 

NUM2=NUM2+50 

NUM3=NUM3+50 



SECOND 'IF* STATEMENT: NUMBER INSIDE PAREN MUST BE 
CNE MULTIPLE OF 50 +1 LOWER THAN Nl; E.G., IF Nl=1286, 
NUMBER INSIDE PAREN SHOULD BE 1251; IF Nl=126, 

NUMBER INSIDE PAREN SHOULD BE 101. 

IF(NUM3.LT.1151) GC TO 199 
NUM2=1151 
NUM3=1190 
GC TO 199 
221 A1=IA1 
A2=IA2 
V=I V 

209. WRITE(6,210) R2,R1,Z 
210 FORMAT! ' 1* , 'CORRELATION AND TESTS'//' ', 

1' R(PATTERN) E OU A L S ' , F 1 5 . 7 , / , 

2 'R(PREDICTOR) ECU AL S ' , F 1 5 . 7 , / , 

3* Z EQUALS' ,F15.7/ ) 

L=1 

GO TO 30 

90 WRITE(6,212) R1,Z 

212 FORMAT! ' O' , 'R( TOTAL CORRECT) E QUA LS ' , F 1 5 . 7 / 

1' Z EQUALS', F15. 7/ ) 



THE VARIABLE R7 IS THE CORRELATION COEFFICIENT 
BETWEEN PATTERN SCORES AND TOTAL CORRECT (ONES) 

R7=N1*UU-I A1=*^X1 
WRITE(6,37) R7,UU 
37 FORMAT!' R7= ' , F 1 8. 4 , ' UU= ' , F 1 8. 4 ) 
R1=(N1*A2)-=(A1^A1) 
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Cross-Validation Listing 
(Continued) 



R8=(N1*X2)-(X1*X1 ) 

0=(R1=^R8)=<‘*. 5 
R7=R7/0 

WRITE(6»213) R7 

213 FORMAK ‘ O' » *R( PATTERN SCORE/TOTAL ONES) E OUALS • , F6 .3 ) 



RMUL IS THE MULTIPLE CORRELATION COEFFICIENT 
TO BE USED IN THE DETERMINATION OF 'F*. 



RMUL = ( (R2**2) + {RlSTAR=^=S^2)-2^(R2=fRlSTAR=^'R7) )/( 1-R7^=^'2) 

RMUL = RMUL>!-^;^.5 

FF = { ( RMUL=t«2)-( R1STAR*>‘'2) ) =«' ( N1 -3 ) / ( 1- (RMUL=5'=^2 ) ) 
WRITE(6,417) RMULtFF 
A17 FORMAT! • O' R( MULT. CORREL. COEF.) EQUALS * , AX, F6.4, //, 
1' FF EQUALS' ,F 8.4) 

STOP 

END 

//G0.FTC6F001 DD SPACE = ( C YL , ( 5 , 1 ) ) 

//GC.FT07F001 CD SYSOUT=B 

//GO.FT04F001 DD DSN = S 05 7 5 . K P W3-, UN IT = 2321 , VOL = S ER=CEL00 1 , 
y/ DCB=(RECFM=FB,BLKSLZE=2000, LRECL=80) ,DI SP=(CLD,KEEP) 
//GG.SYSIN DO 
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Output III - Cross-Validation 

(First 50 SuIAects) 

ETST Bxm 



IDENT 


PREDICTOR 


CRITERION 


1 PATTERN SCORE 


1 TOTAL ONES 


2 


58 


62 


51.3182' 


1 


4 


72 


76 


71.2954 


6 


6 


65 


67 


58. 5000 


3 


8 


59 


52 


52. 5238 


1 


10 


62 


61 


62.2308 


4 


12 


71 


76 


73 .2857 


7 


14 


66 


67 


64.4000 


4 


16 


63 


62 


65.3000 


5 


18 


48 


36 


64.2500 


4 


20 


50 


52 i 


52.5833 


1 


22 


63 


66 I 


66.3333 


5 


24 


61 


53 


54.5500 


1 


26 


50 


56 


57.7308 


2 


28 


59 


65 


54.7500 


2 


30 


62 


64 


64.4000 


4 


32 


61 


58 


54,7500 


2 


34 


71 


68 


65.6097 


5 


36 


65 


62 


62. 8621 


3 


38 


57 


58 


59.7143 


2 


40 


66 


62 


57.7308 


2 


42 


65 


58 


65. 8571 


4 


44 


61 


56 


52. 5833 


1 


46 


68 


66 


69.6667 


5 


48 


70 


75 


73.2857 


7 


50 


67 


59 1 


52.5238 


1 


52 


62 


61 


64..8000 


4 


54 


68 


68 


65.6097 


5 


56 


66 


65 


61.0000 


3 


58 


61 


65 


71.2954 


6 


60 


59 


54 


60.7333 


3 


62 


54 1 


55 


62.8621 


3 


64 


55 ^ 


52 


62.8621 


3 


66 


51 


58 


60. 7333 


3 


68 


56 


5 1 


49.6071 


0 


70 


62 


57 


54.3 000 


2 


72 


60 


61 


62.230S 


4 


74 


73 


67 


68.7500 


6 


76 


65 


7A 


73 .2357 


7 


78 


52 j 


5i 


49.0000 


2 


80 


63 1 


• 74 


66.7667 


5 


82 


66 


63 


54, OCOO 


4 


84 


70 


72 


67.7273 


5 


86 


63 


64 


64.4000 


4 


88 


55 I 


51 


54.5500 


1 


90 


67 


69 


67.7273 


5 


92 


70 


71 


68.7500 


6 


94 


67 


76 


67.7273 


5 


96 


• 53 


64 


65.0000 


4 


98 


60 


62 


57.3913 


2 


100 


72 


62 


67. 7273 


5 
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APPENDIX L 



GLOSSARY OF 
COK4PUTER VARIABLES 
USED IN THESE PROGRAMS 

A(J) - subject’s GOT score used as a predictor 

A1 - sum of predictors 

A2 - sum of squares of predictors 

B(J) - subject's decimal value of his binary score 

C(J) - j_^ subject's final school grade used as the criterion 

Cl - sum of the criterion scores 

02 - sum of squares of criterion score 

D(J) - j_^ subject's identification number 

E(J) - j_^ subject's total correct 

F(,) - joint frequency distribution 

G( ) binary pattern (2 replaced O in output) 

H( ) total correct in a binary pattern 
lAl - sum of total correct (total ones) 

IA2 - sum of square of total correct (total ones) 

IHI 

- used in calculating range of criterion scores 

ILO 

IV - sum of C(J)*E(J) 

IW( )- a column on a subject's record card 
KDNUM - card number 
M - column in 'F matrix’ 
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Glossary of Computer Variables Used in These Programs 

(Continued) 



N 

N1 - 
N2 - 
N3 - 
N4 - 

P(,) - 
R1 - 

R2 - 
R3-5- 
R7 - 
RMUL 
S(I) - 

51 - 

52 - 
V - 
W - 
W(l-7) 
X(J) - 
XI - 
X2 - 



row in *F matrix' 
sample size 

number of elements in the binary pattern 
range of criterion scores 

2**N2; number of combinations of patterns of 1/0 with N2 
questions 

j th subject's pattern of ones/zeros 

correlation coefficient between criterion/predictor 

2nd time corelation coefficient between criterion/total correct 

correlation coefficient between criterion /pattern 

correlation coefficients used in determining R1 and R2 

correlation coefficients between pattern scores and total ones 

- multiple correlation coefficient 

a pattern score associated with a particular pattern 

weighted sum of people with that pattern (weights being the 
criterion scores) 

number of people with that pattern 
sum of product of C(J)*A(J) 
sum of product of C(J)*X(J) 

- an array of questions being used in this study 
j th subject's pattern score 

sum of pattern scores 

sum of square of pattern score 
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Glossary of Computer Variables Used in These Programs 

(Continued) 



Z - test statistic 

F - test statistic F distribution 
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